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(54) Multimedia search and browsing method using multimedia user profile 



(57) The present invention relates to a multimedia 
search and browsing method using a user profile infor- 
mation structure which is capable of displaying search 
items in order of user preference on the basis of item 
ordering criteria according to user preference by using 
search item ordering criteria information and a user pro- 
file including user preference information about the item 
ordering criteria, and browsing a searched object after 
searching the search object indicated by the search 



items displayed according to the user preference. The 
present invention can be adapted to multimedia service 
systems such as a digital VTR (Video Tape Recorder), 
a VOD (Video On Demand), an AOD (Audio On De- 
mand), the internet. 

In addition, the user profile for searching andbrows- 
ing the multimedia according to the present invention 
gets search and browsing of the multimedia performed 
easier in accordance with user's liking. 
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Description 

BACKGROUND OF THE INVENTION 

1 . Field of the Invention 5 

[0001] The present invention relates to a multimedia 
. search and- browsing method using a. multimedia user 
profile information structure, in particular to the multime- 
dia search and browsing method using the multimedia *0 
user profile information structure which is capable of dis- 
playing search items in accordance with user prefer- 
ence, and searching and browsing a pertinent multime- 
dia by using the search items displayed according to the 
user preference. 15 

2. Description of the Prior Art 

[0002] Conventionally a user can watch television by 
receiving a video and an audio signal from a broadcast- 20 
ing station, and can watch a movie in a theater. 
[0003] However, the usermay want to watch a certain 
segment of a drama or movie, accordingly in order to 
make the user select a certain segment of the drama or 
movie, related technologies have been presented ac- 25 
cording to development of various video mediums. 
[0004] For example, the user can choose a movie or. 
a drama selectively by connecting to a video provider 
through a network such as a LAN (Local Area Network). 
[0005] In addition, the user can selectively reproduce so 
a certain moving picture segment in the selected drama 
or movie which shows a character, an accident, a place, 
or an article the user wants to watch. 
[0006] In addition, the user can automatically sum- 
marily reproduce important parts of video contents such 35 
as a news program, a sports program, a movie without 
reproducing the whole, and can reproduce custom-tai- 
lored moving picture segment by setting key frames rep- 
resenting each moving picture segment and displaying 
the key frames. 40 
[0007] There are certain technologies to perform 
above-mentioned function. 

[0008] For example, there is a technology which is ca- 
pable of selecting an object in lists, and reproducing a 
certain segment showing the selected object when the -5 
user wants to watch by dividing various video contents 
into the lists including groups such as a pattern group, 
an index group, "a medium group, a segment group, an 
object group, a representation group and displaying in- 
formation about the selected object such as a character 50 
and an article (object name, position on a screen, nu- 
merical value of an object on the video contents ect.) in 
the object group and representation group. 
[0009] There is the other technology which is capable 
of providing lists about various additional information of 55 
video contents which can be made before, during, after 
production of the video contents. 
[0010] For example, the additional information lists 



can provide information related to positions where char- 
acters and articles show in the video contents, and it can 
produce the pertinent scene when the user selects an 
object. 

[001 1] In addition, when the selected object is an ar- 
ticle, it displays information about the article (manufac- 
turer, price ect.), and connects to the manufacturer/ 
sales agency of the article. 

[0012] There is another technology which is capable 
of providing a partial map having information about each 
part of video contents. 

[0013] In other words, the partial map displays the in- 
formation about the each part of the video contents 
(items : violence, suggestiveness. importance, charac- 
ters, positions, difficulty ect.), the user can reproduce 
the wanted part among the video contents by registering 
user preference about the items of the video partial map, 
and can restrict transmission of a certain content to un- 
identified users. 

[0014] There is another technology which is capable 
of illustrating a time connection graph of moving picture 
shots. 

[0015] However, it is difficult to understand a story of 
the time connection graph of the moving picture shots, 
because it is as same as watching a few representative 
scenes. 

[0016] The above-described conventional moving 
picture search technologies just display lists informing 
various information about the objects showing in the 
drama or movie, and perform the function according to 
selection of the user. 

[0017] In addition, it is possible to search and browse 
the multimedia efficiently because the user can get the 
part of the multimedia related to the selected item. 
[0018] However, search and browsing itemsi have to 
be increased in accordance with increase of the number 
of multimedia. Accordingly technologies for displaying 
selectively search items in accordance with user liking 
are required, and are developed. 
[0019] For example, there is an automatic setting 
method which is capable of setting a system automati- 
cally when the user approaches to the system by storing 
a television channel, a music genre, volume of the music 
preferred by the user and using the stored information. 
[0020] There is the other method which is capable of 
searching a program in accordance with user liking by 
comparing a program character table storing informa- 
tion such as characters, program genre, category of 
sports, sports team ect. with a user preference table. 
[0021] There is another method which is capable of 
searching goods corresponding to a request of the user 
by comparing a stored character (size, price, capability 
ect.) of the goods with the character requested by the 
user. 

[0022] There is another method which is capable of 
setting automatically parameters of a video, display unit 
such as a volume, brightness by learning preference of 
the user. 
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[0023] There is another method which is capable of 
displaying e-mails in accordance with user liking when- 
ever the user checks the e-mails. 
[0024] There is another method which is capable of 
providing a custom-tailored electronic newspaper by 5 
learning preference of the user. 
[0025] The above-described multimedia search and 
browsing systems store preference about search items 
to be searched/browsed, and search/browse the multi- 
media in accordance with the preference. io 
[0026] In the above- systems a searcher and a brows- 
er display search items (character, place, incident, arti- 
cle, key frame ect.) to the user, there are many methods 
to display the search items to the user. 
[0027] For example, the search items can be dis- ^ 
played according to various criteria such as alphabetical 
order, number of appearance, priority, occurrence order 
ect.. 

[0028] The each user can have different preference 
about the criteria, but the conventional technologies did 20 
not consider the preference difference in the criteria. 

SUMMARY OF THE INVENTION 

[0029] The object of the present invention is to provide 25 
a multimedia search and browsing method using a mul- 
timedia user profile information structure which is capa- 
ble of displaying search items in accordance with pref- 
erence of a user and searching and browsing a pertinent 
multimedia by using the displayed .search items in ac- 30 
cordance with the preference. 

[0030] In order to achieve above-mentioned object, 
the multimedia user profile information structure for in- 
dexing and browsing a multimedia object comprises 
search item ordering criteria information for searching 35 
and browsing the multimedia object, and user prefer- 
ence information on the search item ordering criteria for 
displaying the search item to be browed in an userpre- 
ferred arrangement. 

[0031] In order to achieve above-mentioned object, a *o 
search item preference information structure for search- 
ing and browsing a multimedia comprises a search ob- 
ject which is the multimedia data stream, a search item 
which is criterion of indexing, connection information 
which connects the each search item to the search ob- 45 
ject. ordering criteria information for informing ordering 
criteria of the each search item, a multimedia object in- 
cluding ordering information according to the each or- 
dering criterion, item categories for informing items 
which can be criteria of the search and browsing, a user 50 
profile including preference value informing preference 
about the ordering information of the multimedia object 
on the each item category, and displays the each search 
item ordering in accordance with the each user's liking. 
[0032] In order to achieve above-mentioned object, a 55 
multimedia search and browsing method using multime- 
dia user profile information for indexing and browsing a 
multimedia object comprises a display process for dis- 



playing search items in order of user preference on the 
basis of the item ordering criteria according to the user 
preference by using the search item ordering criteria in- 
formation and user profile including the user preference 
information about the item ordering criteria, and a 
browsing process for browsing a searched object after 
searching the search object indicated by the search 
items displayed according to the user preference. 
[0033] In order to achieve above-mentioned object, a 
multimedia search and browsing method using the user 
profile information structure" for indexing and browsing 
a multimedia object comprises a search object which is 
the multimedia data stream, a search item which is cri- . 
terion of indexing, connection information which con- 
nects the each search item to the search object, order- 
ing criteria information for informing ordering criteria of 
the each search item, a multimedia object including or- 
dering information in accordance with the each ordering 
criterion, item categories for informing items which can 
• be criteria of the search and browsing, a user profile in- 
cluding preference value informing preference about the 
ordering information of the multimedia object on the 
each item category, a selecting process for selecting the 
ordering criteria information in order of the preference 
value of the user profile on the each search item in 
search and browsing of the multimedia, a displaying 
process for displaying the multimedia items by using the 
ordering criteria information of the user profile, and a 
browsing process for browsing. the search object after 
searching the search object indicated by the search 
items displayed according to the user preference. 

BRIEF DESCRIPTION OF THE DRAWINGS 

[0034] FIG.1 illustrates a multimedia data structure 
according to the embodiment of the present invention. 
[0035] FIG.2 illustrates the other embodiment of the 
multimedia data structure in FIG.1 . 
[0036] FIG.3A-3D illustrate the other embodiment of 
the multimedia data structure in FIG.1. 
[0037] . FIG:4 illustrates a structure of search item or- 
dering criteria preference information according to the 
present invention. 

[0038] FIG. 5 illustrates a search item preference in- 
formation structure according to the present invention. 
[0039] FIG. 6 illustrates the embodiment of the search 
item preference according to the present invention. 
[0040] FIG. 7 illustrates a video browser using the user 
preference in FIG. 6. 

[0041] FIG.8A-8B illustrate ordering information of a 
multimedia object in accordance with the ordering crite- 
ria information. 

[0042] FIG. 9 illustrates a structure of search item or- 
dering criteria preference information by using the mul- 
timedia object in FIG. 8. 
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DETAILED DESCRIPTION OF THE PREFERRED 
EMBODIMENT 

[0043] FIG.1 illustrates a multimedia data structure 
according to the embodiment of the present invention. 
[0044] As depicted in FIG.1 , a multimedia object 1 01 
comprises a search item 1.02 for searching and brows- 
ing, and a search object 103 for being searched and 
browsed. 

[0045] The search item 102 comprises category of 
item 104 informing the categories of the item, and a 
search object connection, information 105 for connect- 
ing to the search object. 

[0046] When the multimedia object 1 01 is moving pic- 
ture, items comprised in the search item 102 can be a 
character or an incident. 

[0047] The search object 103 can be a segment 
showing the character or incident, and the search object 
connection information 105 can be information for con- 
necting a pertinent character and incident to a pertinent 
segment. 

[0048] FIG.2 illustrates the other embodiment of the 
multimedia data structure in FIG.1 . 
[0049] First, an audio-video description scheme 201 
comprises a syntactic DS (Description Scheme) 202 for 
describing a segment showing the multimedia object, 
and a semantic DS (Description Scheme) 203 for de- 
scribing meaningful information. 
[0050] The syntactic DS (Description Scheme) 202 
comprises a segment DS (Description Scheme) 204 as 
a real moving picture segment, a region DS (Description 
Scheme) 205, and a segment/region relation graph 206. 
[0051] The semantic DS (Description Scheme) 203 
comprises an object DS (Description Scheme) 207 for 
describing object information such as a character or a 35 
place, an event DS (Description Scheme) 208 describ- 
ing incidents, and an object/event relation graph 209 for 
describing relation between the characters, places or in- 
cidents. 

[0052] Herein, the search item 102 comprises the 
segment DS (Description Scheme) 204. region DS (De- 
scription Schemgf205, segment/region relation graph 
206. object DS (Description Scheme) 207, event DS 
(description Scheme) 208, object/event relation graph 
209, and the category of item 104 divides the above 
search items. 

[0053] FIG.3A-3D illustrate the other embodiment of 
the multimedia data structure in FIG.1 . 
[0054] In other words, FIG.3A-.3D illustrate examples 
of the category of item 1 04 and search object 1 03 of the 
multimedia object 101 in FIG.1. FIG.3A-3D will be de- 
scribed in detail with reference to FIG.1 . 
[0055] First, the category of item 1 04 can be all search 
items such as a character, key frame, place, actor/ac- 
tress, article, keyword, melody ect.. 
[0056] The search objecf 1 03 can be a moving picture 
segment showing a character, moving picture segment 
represented by a key frame, a movie or a drama show- 



ing a certain actor/actress, a document comprising a 
certain key word, or an audio clip having a melody sim- 
ilar to a certain melody ect.. 

[0057] FIG.3A illustrates an ordinary text document 
item, the text document is a key word in the category of 
item 104, and the search item 102 can be key word 1 , 
key word 2, key word 3,..., key word N. 
[0058] The search object 103 can be document 1, 

document 2. document 3 document N. 

io [0059] • FIG.3 illustrates moving picture item catego- 
ries, the item category is characters, the search item is 

character 1 , character 2. character 3 character N. 

[0060] In addition the .search object 103 can be ap- • 
j pearance segment N showing a pertinent character. 
13 [0061] FIG.3C illustrates the other example of moving 
picture item categories, the category of item 1 04 is key 
frame, the search item 1 02 can be key frame 1 key frame 

2. key frame 3 key frame N. 

[0062] The search object 103 can be appearance 
20 segment i, appearance segment 2. appearance seg- 
ment 3, ... , appearance segment N. 
[0063] Fig.3D illustrates the other example of moving 
picture item categories, the category of item 1 04 is actor ■ 
name, the search item 1 02 can be actor 1 . actor 2, actor 
23 3, ... , actor N. 

[0064] The search object 1 03 can. be movie 1 , movie 

2, movie 3. ... , movie N showing the pertinent actor. 
[0065] FIG. 4 illustrates a structure of search item or- 
dering criteria preference information according to the 

30 present invention. 

[0066] In other words, it describes a user profile infor- 
mation structure comprising preference of the search 
item ordering criteria informing a user preference about 
•the multimedia object structure such as FIG.1. or FIG. 

3, it will now be described in detail. 
[0067] First, a search item ordering criteria preference 
401 comprises a category of search item 402," and an 
item ordering criteria 403 determining the ordering of the 
search items according to the category of item. 

40 [0068] The item ordering criteria 403 comprises a cat- 
egory of criteria 404, and a preference value 405 about 
each category of criteria. 

[0069] Accordingly, the multimedia object in FIG. 1 . or 
FIG.3 can be searched by using the user profile infor- 
ms mation in FIG.4 informing the search item ordering cri- 
teria. •:. . 

[0070] In addition, in browsing of the multimedia ob- 
ject, the search item can be displayed by using the 
search item ordering criteria in accordance with the user 
50 preference. 

[0071] Herein, when the user searches and browses 
a certain object, a searcher displays the search items to 
the user The search items ordering is performed in ac- 
cordance with the item ordering criteria 403 of the user 
55 profile information such as depicted in FIG. 4. 

[0072] For example, when a character of moving pic- 
ture is searched and browsed, the item ordering criteria 
403 can be number of appearance of the character or 
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age of the character, the search/browsing is performed 
by using a key frame of the moving picture, the item or- 
dering criteria 403 can be time ordering of the moving 
picture represented by the key frame, brightness of the 
key frame image, or color of the key frame image. 
[0073] In addition, the preference value 405 of the 
item ordering criteria 403 is information stored by the 
user. 

[0074] When the user selects preferred criteria, stores 
it among the item ordering criteria 403, and uses the 
searcher for searching and browsing, the searcher dis- 
plays the search items on the basis of the preference 
value of the user about the item ordering criteria. 
[0075] FIG. 5 illustrates a search item preference in- 
formation structure according to the present invention. 
[0076] As depicted in Fig. 5, a search item preference 
501 comprises a -search item 502, a category of item 
503, a item preference 504, and a preference value 505. 
[0077] The category of item 503 can be characters, 
places, and key frames as described above. 
[0078] The search item 502 can be character N when 
the category of item is character. 
[0079] The item preference 504 describes the user 
preference about a certain search item among the cat- 
egory of item such as the character, place, key frame 
ect. as the preference value 505. 
[0080] FIG. 6 illustrates the embodiment of the search 
item preference according to the present invention. 
[0081] As depicted in FIG. 5, when a category of item 
is character, the user preference is described as a cer- 
tain value on the each search item of the each character 
(character N). 

[0082] For example, the character 1 can be prefer- 
ence '1 00' , character 3 can be preference '46', the pref- 
erence value is displayed on the each search item (char- 
acter). 

[0083] In other words, the multimedia searcher and 
browser can display the characters as character 1 , char- 
acter 2, character 3 character N from the user pro- 
file having the preference information. 
[0084] In addition, the user can easily search/browse 
the segments showing the pertinent character by watch- 
ing the search item displayed in order of the preference 
of the user. 

[0085] FIG. 7 illustrates a video browser using the user 
preference in FIG. 6. 

[0086] In other words, it depicts a moving picture 
browser displayed on the basis of characters. 
[0087] As depicted in FIG. 7, the search item is dis- 
played on a character screen 701 from character 1 to 
character N (N=4) in order of the preference of the user. 
[0088] Herein, when the character 1 is selected by the 
user, the key frames corresponding to the moving pic- 
ture (search object) showing the character 1 are dis- 
played on a major scene screen 702. Herein, a main 
screen 703 reproduces a pertinent moving picture seg- 
ment when the user selects a scene 3. 
[0089] FIG.8A-8B illustrate ordering information of a 



multimedia object in accordance with the ordering crite- 
. ria information, 
[0090] • FIG. 8A illustrates an object DS comprising ob- 
ject information 802 and ordering information 803. 
^ [0091] FIG 8B illustrates the ordering information on 
the basis of three ordering criteria information (priority, 
age, name), ' 

[0092] in other words, when the object DS (Descrip- 
tion Scheme) 801 is a character in the multimedia struc- 

10 ture described in FIG.2, the object DS (Description 
Scheme) 801 comprises the object meaning information 
802 displaying the object type and object name, and the 
ordering information 803 displaying the ordering criteria 
information' and ordering Key. 

is [0093] Herein, the object DS (Description Scheme) 
displays the ordering information in the basis of the three 
ordering criteria information such as a priority, age, 
name. 

[0094] As depicted in; FIG.8B, the characters A1 , T2, 
20 T1, B2,,, are displayed in accordance with the priority, 

the character B2, A1 , B2,,„ are displayed in accordance 

with the age, and the character A1. A2, A3, B1 1M1 are 

displayed in accordance with the name. 

[0095] FIG. 9 illustrates a structure of search item or- 
2 5 dering criteria preference information by using the mui- ' 

timedia object in FIG. 8. 

[0096] As depicted in FIG. 9, a search item ordering 
criteria preference 901 comprises- a key item 902, and 
an ordering Preference 903. . 

30 [0097] The ordering - preference 903 comprises a ref- 
erence to criteria 904, and a preference value 905. 
[0098] The reference to criteria 904 indicates the or- 
dering criteria information displayed on the multimedia 
object, and the preference value 905 means the prefer- 

35 ence about the referenced ordering criteria. 

[0099] Accordingly, a search item ordering based on 
the user preference and the multimedia search and 
browsing using the search item ordering can be possible 
by referencing the user profile information in FIG. 9 from 

40 the multimedia object in FIG. 8. 

[0100] In other words, the reference to criteria 904 
based on the ordering criteria preference value 905 dis- 
plays search items of the multimedia object on the basis 
of the ordering criteria information of the multimedia ob- " 

4 5 ject data structure in FIG. 8 by using the search item or- 
dering criteria preference 901 described in the user pro- 
file. 

[0101] After that, the search and browsing about the 
pertinent multimedia object are performed from the dis- 

50 piayed information. 

[0102] As described above, when the search items re- 
lated to the multimedia are displayed, the present inven- 
tion can efficiently perform search and browsing of the 
multimedia object by displaying the search item after se- 

55 lecting a criterion among search item ordering criteria in 
accordance with the user preference. 
[0 1 03] In addition, the present invention can be adapt- 
ed in a multimedia service system such as a VTR (video 



9 



EP 1 091 584 A2 



10 



Tape Recorder), a VOD (Video On Demand), an AOD 
(Audio On Demand), the internet ect.. 
[0104] In addition, the present invention is capable of 
searching/browsing easily the object according to pref- 
erence of the user. 5 



Claims 

1. A multimedia user profile information structure for w 
indexing and browsing a multimedia object, 
comprising : 

. search item ordering criteria information for 
searching and browsing the multimedia object; ?5 
and 

user preference information on the search item 
ordering criteria for displaying the search item 
to be browed in an userpreferred arrangement. 

20 

2. The multimedia user profile information structure 
according to claim 1 , wherein the user profile com- 
prises item categories which can be criteria of 
search and browsing in order to get each search ' 
and browsing item ordering displayed according to 25 
each user's liking. 

3. The multimedia user profile information structure 
according to claim 2, wherein the userprofile further 
comprises item ordering criteria in order to deter- 30 
mine ordering of search items on the each item cat- 
egory. 

4. The multimedia user profile information structure 
according to claim 3, wherein the item ordering cri- 35 
teria comprise criteria categories informing the cat- 
egories of the criteria. 

5. The multimedia user profile information structure 
according to claim 4, wherein the item ordering cri- io 
teria further comprise preference information in- 
forming preference about the pertinent criterion cat- 
egory. ' 

6. The multimedia user profile information structure *s 
according to claim 4, wherein the.criteria categories 
inform the preference of the user file about the per- 
tinent item, and the search items are displayed in 
accordance with the preference of the user profile. 

50 

7. A multimedia search and browsing method using 
multimedia user profile information for indexing and 
browsing a multimedia object, comprising : 

displaying search items in order of user prefer- 55 
ence on the basis of the item ordering criteria 
according to the user preference by using the 
search item ordering criteria information and 



userprofile including the user preference infor- 
mation about the item ordering criteria ; and 
browsing a searched object after searching the 
search object indicated by the search items dis- 
played according to the user preference. 

8. A search item preference information, structure for 
searching and browsing a multimedia, comprising : 

a search object which is the multimedia data 
stream ; 

a search item which is criterion of indexing ; 
connection information which connects the 
each search item to the search object ; 
ordering criteria information for informing or- 
dering criteria of the each search item ; and 
a multimedia object including ordering informa- 
tion according to the each ordering criterion. 

9. The search item preference information structure 
for searching and browsing the multimedia accord- 
ing to claim 8, wherein the search item preference 
information structure for searching arid browsing 
the multimedia further comprises : 

item categories for informing items which can 
be criteria of the search and browsing ; and 
a user profile including preference value in- 
forming preference about the ordering informa- 
tion of the multimedia object on the each item 
category. 

10. The search item preference information structure 
for searching and browsing the multimedia accord- 
ing to claim 8 or 9, wherein the search item prefer- 
ence information structure for searching and brows- 
ing the multimedia makes the each search item or- 
dering display according to each user's liking. 

1 1 . A multimedia search and browsing method using a 
user profile information structure for indexing and 
browsing a multimedia object, comprising : 

a search object which is the multimedia data 
stream ; 

a search item which is criterion of indexing ; 
connection information which connects the 
each search item to the search object ; 
ordering criteria information for informing orr - 
dering criteria of the each search item ; 
a multimedia object including ordering informa- 
tion in accordance with the each ordering 
criterion ; 

item categories for informing items which can 
be criteria of the search and browsing ; 
a user profile including preference value in- 
forming preference about the ordering informa- 
tion of the multimedia object on the each item 
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category ; 

selecting the ordering criteria information in or- 
der of the preference value of the user profile 
on the each search item in search and browsing 
of the multimedia ; 5 
displaying the multimedia items by using the or- 
dering criteria information of the user profile ; 
and 

browsing the search object after searching the 
search object indicated by the search items dis- 10 
played according to the user preference. 
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